System and method for mixed reality

ABSTRACT

A mixed reality system, comprising: a data acquisition device configured to acquire real-world data; an output device for providing the real-world data to a user; and a processing circuitry configured to: obtain (a) data acquired by the data acquisition device, and (b) information of one or more virtual entities having properties enabling determination of simulated effects of the virtual entities on the data; determine the simulated effects of the virtual entities on the data utilizing the properties; and provide the user with output on the output device being a manipulation of the data reflecting the simulated effects.

TECHNICAL FIELD

The invention relates to a system and method for mixed reality.

BACKGROUND

Mixed reality is the merging of real and virtual worlds to produce newenvironments and visualizations where physical and digital objectsco-exist and interact in real-time. Mixed reality takes place not onlyin the physical world or the virtual world, but is a hybrid of realityand virtual reality via immersive technology.

Mixed reality is used for a number of different applications:entertainment (e.g. interactive games, interactive movies, etc.),simulation-based learning, simulation-based to training, where realityis simulated and represented in complex, layered data through an outputdevice.

In some cases, the mixed reality environment includes a physical sensor(e.g. a radar, a camera, an infrared sensor, etc.) that displays, on anoutput device of the sensor (e.g. a screen, a Head Mounted Display,etc.), a combination of real-world visualizations of real-world data(acquired by the sensor) and virtual entity visualizations (locatedwithin an area from which the real-world data readings are acquired).The real-world visualizations can be affected by one or more parametersof the sensor (e.g. noise model characterizing readings of the sensor,spectral sensitivity of the sensor, signal-to-noise ratio of the sensor,etc.) or by one or more algorithms, employed on the readings of thereal-world data by the sensor or by a communication link that is used tocommunicate the data from the location of the sensor itself to thelocation of the output device (e.g. a sensor is carried on a drone andthe real-world data readings acquired by the sensor are communicated viaa communication link to an output device located in a Ground ControlStation (GCS) that is remote from the drone carrying the sensor).

A non-limiting example is an optical sensor with a fisheye lens thatdisplays real-word visualizations with a visual distortion intended tocreate a wide panoramic or hemispherical image.

Current mixed reality systems, that include a physical sensor, do nottake into consideration (a) the effects of the one or more parameters ofthe sensor on the virtual entity visualizations, (b) the effects of theone or more algorithms on the virtual entity visualizations, employed onthe readings of the real-world data by the sensor and (c) the effects ofthe communication link between the physical sensor and the output deviceon the virtual entity visualizations. In the current mixed realitysystems, the virtual entity visualizations are displayed by (a)injecting the virtual entity visualizations into the output device,and/or (h) by displaying the virtual entity visualizations on asee-through output device overlaying the output device, and/or (c) byprojecting the virtual entity visualizations on the real-worldvisualization. Thus, the virtual entity visualizations are not affectedby the one or more parameters of the sensor, by the one or morealgorithms or by the communication link as the real-world data acquiredby the sensor. In these cases, the virtual entity visualizations areprovided to the output device through a communication channel that isdifferent from the communication channel through which the real-worldvisualizations are provided to the output device, thus the virtualentity visualizations are not affected as the real-world visualizationsare affected.

This causes current mixed reality simulation systems that include aphysical sensor to display the virtual entity visualizations, on theoutput device, in a way that is perceived to be different than thereal-world visualizations. Continuing our non-limiting example: theoptical sensor with the fisheye lens will display real-wordvisualizations with a fisheye visual distortion, but virtual entityvisualizations will be displayed without the fisheye visual distortion,thus a user of the optical sensor will be able to discern between thereal-word visualizations and the virtual entity visualizations displayedon the output device of the optical sensor—an outcome that can beproblematic in mixed reality environments where the user should notperceive virtual entities differently than the real-world data.

In addition, current mixed reality simulation systems that include adata acquisition device (e.g. a physical sensor, a radio receiver, etc.)do not change the way the acquisition device displays the acquired datain a mixed reality environment in response to an effect by a virtualeffect. For example, a radar, that is part of a mixed reality simulationsystem can have temporary restricted coverage because of a simulatedvirtual interference, however, in current mixed reality simulationsystems will display the acquired data without taking into account thetemporary restricted coverage.

Another non-limiting example can be a thermal sensor that should beblinded (or otherwise affected) by virtual heat simulated in an areaviewed by the sensor, but current mixed reality simulation systems willnot take into account an effect of the virtual heat on the thermalsensor. Another example can be a real-world radio receiver that wouldnot have been able to receive radio signals in a given channel due to avirtual radio source transmitting it's a virtual radio signal in thegiven channel, but current mixed reality simulation systems will nottake into account an effect of the virtual radio source.

There is thus a need in the art for a new method and system for mixedreality.

References considered to be relevant as background to the presentlydisclosed subject matter are listed below. Acknowledgement of thereferences herein is not to be inferred as meaning that these are in anyway relevant to the patentability of the presently disclosed subjectmatter.

U.S. Pat. No. 8,616,884 (Lechner et al.) published on Dec. 31, 2013,discloses a method and apparatus for training in an aircraft. A displaysystem is associated with an aircraft. A sensor system is associatedwith the aircraft. A training processor is configured to be connected tothe aircraft. The training processor is configured generate constructivedata for a number of simulation objects and generate simulation sensordata using the constructive data. The training processor is furtherconfigured to present the simulation sensor data with live sensor datagenerated by the sensor system for an aircraft on a display system inthe aircraft.

US Patent application No. 2017/0294135 (Lechner) published on Oct. 12,2017, discloses a system is provided for real-time, in-flight simulationof a target. A sensor system may generate a live stream of anenvironment of an aircraft during a flight thereof, the live streamhaving associated metadata with structured information indicating areal-time position of the aircraft within the environment. A targetgenerator may generate a target image from a source of information fromwhich a plurality of different target images may be generable. Thetarget generator may also generate a synthetic scene of the environmentincluding the target image. A superimposition engine may thensuperimpose the synthetic scene onto the live stream such that thetarget image is spatially and temporally correlated with the real-timeposition of the aircraft within the environment. The live stream withsuperimposed synthetic scene may be output for presentation on a displayof the aircraft during the flight.

US Patent application No. 2013/0286004 (McCulloch et al.) published onOct. 31, 2013, discloses a technology described for displaying acollision between objects by an augmented reality display device system.A collision between a real object and a virtual object is identifiedbased on three-dimensional space position data of the objects. At leastone effect on at least one physical property of the real object isdetermined based on physical properties of the real object, like achange in surface shape, and physical interaction characteristics of thecollision. Simulation image data is generated and displayed simulatingthe effect on the real object by the augmented reality display. Virtualobjects under control of different executing applications can alsointeract with one another in collisions.

US Patent application No. 2018/0075658 (Lanier et al.) published on Mar.15, 2018, discloses techniques that include mixed reality tools,referred to as HoloPaint, that allow use of any of a variety of sensorsto determine physical parameters of real objects in a mixed realityenvironment. HoloPaint may correlate current measurements of the realworld with past measurements to perform inventory management, analysisof changes of physical parameters of real objects and environments, andso on. A user may select which parameter to analyze by selecting aparticular type of virtual paint, such as for drawing onto an object tobe analyzed.

GENERAL DESCRIPTION

In accordance with a first aspect of the presently disclosed subjectmatter, there is provided a mixed reality system, comprising: a dataacquisition device configured to acquire real-world data; an outputdevice for providing the real-world data to a user; and a processingcircuitry configured to: obtain (a) data acquired by the dataacquisition device, and (b) information of one or more virtual entitieshaving properties enabling determination of simulated effects of thevirtual entities on the data; determine the simulated effects of thevirtual entities on the data utilizing the properties; and provide theuser with output on the output device being a manipulation of the datareflecting the simulated effects.

In some cases, the data acquisition device is a sensor or a radioreceiver.

In some cases, the sensor is one of the following: a camera, a radar,Night Vision Goggles (NVG), a proximity sensor, temperature sensor, aninfrared sensor, pressure sensor, light sensor, touch sensor, ultrasonicsensor, color sensor, humidity sensor, tilt sensor, accelerometer, or anacoustic sensor.

In some cases, the processing circuitry determines the simulated effectsalso utilizing one or more parameters of the sensor or the radioreceiver.

In some cases, the data is acquired from a training environment, andwherein the virtual entities are designed to simulate trainingscenarios.

In some cases, the simulated effect is one or more of: virtual heat,virtual light, virtual touch, virtual shade, virtual sound, virtualtopography, virtual smoke, virtual hit, or virtual ice.

In accordance with a second aspect of the presently disclosed subjectmatter, there is provided a method comprising: obtaining, by aprocessing resource, (a) data acquired by a data acquisition deviceconfigured to acquire real-world data, and (b) information of one ormore virtual entities having properties enabling determination ofsimulated effects of the virtual entities on the data; determining, bythe processing resource, the simulated effects of the virtual entitieson the data utilizing the properties; and providing, by the processingresource, a user of an output device, used for providing the real-worlddata to the user, with output on the output device being a manipulationof the data reflecting the simulated effects.

In some cases, the data acquisition device is a sensor or a radioreceiver.

In some cases, the sensor is one of the following: a camera, a radar,Night Vision Goggles (NVG), a proximity sensor, temperature sensor, aninfrared sensor, pressure sensor, light sensor, touch sensor, ultrasonicsensor, color sensor, humidity sensor, tilt sensor, accelerometer, or anacoustic sensor.

In some cases, the determining of the simulated effects also utilizesone or more parameters of the sensor or the radio receiver.

In some cases, the data is acquired from a training environment, andwherein the virtual entities are designed to simulate trainingscenarios.

In some cases, the simulated effect is one or more of: virtual heat,virtual light, virtual touch, virtual shade, virtual sound, virtualtopography, virtual smoke, virtual hit, or virtual ice.

In accordance with a third aspect of the presently disclosed subjectmatter, there is provided a non-transitory computer readable storagemedium having computer readable program code embodied therewith, thecomputer readable program code, executable by at least one processor ofa computer to perform a method comprising: obtaining; by a processingresource, (a) data acquired by a data acquisition device configured toacquire real-world data, and (b) information of one or more virtualentities having properties enabling determination of simulated effectsof the virtual entities on the data; determining, by the processingresource, the simulated effects of the virtual entities on the datautilizing the properties; and providing, by the processing resource, auser of an output device, used for providing the real-world data to theuser, with output on the output device being a manipulation of the datareflecting the simulated effects.

In accordance with a fourth aspect of the presently disclosed subjectmatter, there is provided a mixed reality system, comprising: a sensorconfigured to acquire readings of real-world data, and display, on anoutput device, a real-world visualization of the real-world data basedon the readings to a user, wherein the sensor has one or more parametersaffecting the real-world visualization; and a processing circuitryconfigured to: obtain information of one or more virtual entitieslocated within an area from which the readings are acquired, theinformation defining, for each of the virtual entities, one or moresimulated physical properties; determine, for at least one given virtualentity of the virtual entities, a virtual entity visualization of thegiven virtual entity, the virtual entity visualization determined bymanipulating a simulated reading of the simulated physical propertiesbased on the parameters affecting the real-world visualization; anddisplay the virtual entity visualizations in combination with thereal-world visualization, thereby enabling a user viewing the outputdevice to view the virtual entity visualizations and the real-worldvisualization.

In some cases, the display is performed by injecting the virtual entityvisualizations into the output device.

In some cases, the display is performed on a see-through output deviceoverlaying the output device.

In some cases, the display is performed by projecting the virtual entityvisualizations on the real-world visualization.

In some cases, the real-world visualization is provided to the outputdevice via a first communication channel and the virtual entityvisualizations are injected into the output device via a secondcommunication channel other than the first communication channel.

In some cases, the parameters affect the real-world visualization byaffecting the first communication channel.

In some cases, the manipulating of the simulated reading furtherincludes simulating communication effects of the first communicationchannel on the virtual entity visualizations.

In some cases, the virtual entity visualization of the given virtualentity is perceived closer to a third visualization of the given virtualentity had it been a real-world entity having physical propertiesidentical to the simulated physical properties than an alternativevisualization of the given virtual entity determined withoutmanipulating the simulated reading of the simulated physical propertiesbased on the parameters affecting the readings of the sensor.

In some cases, the manipulating of the simulated reading furtherincludes employing one or more algorithms, employed on the readings ofthe real-world data, also on the simulated reading.

In some cases, the parameters affect the real-world visualization byaffecting the readings.

In some cases, the parameters include one or more of: a noise modelcharacterizing readings of the sensor; a spectral sensitivity of thesensor; a spectral response of the sensor; a saturation of the sensor; adynamic range of the sensor; a dark noise of the sensor; asignal-to-noise ratio of the sensor; a detection limit of the sensor; aphoto response non uniformity of the sensor; a penetration depth of thesensor; a lens distortion of the sensor; an optical deformation model ofthe sensor; a vignetting model of the sensor; response to differentexposure levels of the sensor; or a resolution map of the readingsacquired by the sensor.

In some cases, the processing circuitry is further configured to: obtaininformation of movement of the sensor during acquisition of thereadings; determine if the movement of the sensor generates an effect onthe real-world visualization, utilizing the information and theparameters; wherein upon determining that the movement generates theeffect, the manipulating of the simulated reading further includessimulating the effect on the simulated reading.

In some cases, the processing circuitry is further configured to: obtaininformation of simulated movement of the given virtual entity duringacquisition of the readings; determine if the simulated movement of thegiven virtual entity is required to generate an effect on the simulatedreading, utilizing the information and the parameters; wherein upondetermining that the movement is required to generate the effect, themanipulating of the simulated reading further includes simulating theeffect on the simulated reading.

In some cases, the processing circuitry is further configured to: obtaininformation of environmental parameters of at least part of a regionfrom which the readings are acquired; determine if the environmentalparameters generate an effect on the real-world visualization, utilizingthe information and the parameters; wherein upon determining that theenvironmental parameters generate the effect, the manipulating of thesimulated reading further includes simulating the effect on thesimulated reading.

In some cases, the processing circuitry is further configured to: obtainvalues of one or more situational parameters indicative of a state ofthe sensor during acquisition of the readings; and wherein the virtualentity visualization is determined also based on the values of thesituational parameters.

In some cases, the situational parameters include one or more of: atemperature of the sensor; a vibration frequency of the sensor; or atype of platform the sensor is connected to.

In accordance with a fifth aspect of the presently disclosed subjectmatter, there is provided a method comprising: obtaining, by aprocessing circuitry, information of one or more virtual entitieslocated within an area from which readings are acquired by a sensorconfigured to acquire readings of real-world data, and display, on anoutput device, a real-world visualization of the real-world data basedon the readings to a user, wherein the sensor has one or more parametersaffecting the real-world visualization, and wherein the informationdefining, for each of the virtual entities, one or more simulatedphysical properties; determining, by the processing circuitry, for atleast one given virtual entity of the virtual entities, a virtual entityvisualization of the given virtual entity, the virtual entityvisualization determined by manipulating a simulated reading of thesimulated physical properties based on the parameters affecting thereal-world visualization; and displaying, by the processing circuitry,the virtual entity visualizations in combination with the real-worldvisualization, thereby enabling a user viewing the output device to viewthe virtual entity visualizations and the real-world visualization.

In some cases, the display is performed by injecting the virtual entityvisualizations into the output device.

In some cases, the display is performed on a see-through output deviceoverlaying the output device.

In some cases, the display is performed by projecting the virtual entityvisualizations on the real-world visualization.

In some cases, the real-world visualization is provided to the outputdevice via a first communication channel and the virtual entityvisualizations are injected into the output device via a secondcommunication channel other than the first communication channel.

In some cases, the parameters affect the real-world visualization byaffecting the first communication channel.

In some cases, the manipulating of the simulated reading furtherincludes simulating communication effects of the first communicationchannel on the virtual entity visualizations.

In some cases, the virtual entity visualization of the given virtualentity is perceived closer to a third visualization of the given virtualentity had it been a real-world entity having physical propertiesidentical to the simulated physical properties than an alternativevisualization of the given virtual entity determined withoutmanipulating the simulated reading of the simulated physical propertiesbased on the parameters affecting the readings of the sensor.

In some cases, the manipulating of the simulated reading furtherincludes employing one or more algorithms, employed on the readings ofthe real-world data, also on the simulated reading.

In some cases, the parameters affect the real-world visualization byaffecting the readings.

In some cases, the parameters include one or more of: a noise modelcharacterizing readings of the sensor; a spectral sensitivity of thesensor; a spectral response of the sensor; a saturation of the sensor; adynamic range of the sensor; a dark noise of the sensor; asignal-to-noise ratio of the sensor; a detection limit of the sensor; aphoto response non uniformity of the sensor; a penetration depth of thesensor; a lens distortion of the sensor; an optical deformation model ofthe sensor; a vignetting model of the sensor; response to differentexposure levels of the sensor; or a resolution map of the readingsacquired by the sensor.

In some cases, the method further comprising: obtaining, by theprocessing circuitry, information of movement of the sensor duringacquisition of the readings; determining, by the processing circuitry,if the movement of the sensor generates an effect on the real-worldvisualization, utilizing the information and the parameters; whereinupon determining that the movement generates the effect, themanipulating of the simulated reading further includes simulating theeffect on the simulated reading.

In some cases, the method further comprising: obtaining, by theprocessing circuitry, information of simulated movement of the givenvirtual entity during acquisition of the readings; determining, by theprocessing circuitry, if the simulated movement of the given virtualentity is required to generate an effect on the simulated reading,utilizing the information and the parameters; wherein upon determiningthat the movement is required to generate the effect, the manipulatingof the simulated reading further includes simulating the effect on thesimulated reading.

In some cases, the method further comprising: obtaining, by theprocessing circuitry, information of environmental parameters of atleast part of a region from which the readings are acquired;determining, the processing circuitry, if the environmental parametersgenerate an effect on the real-world visualization, utilizing theinformation and the parameters; wherein upon determining that theenvironmental parameters generate the effect, the manipulating of thesimulated reading further includes simulating the effect on thesimulated reading.

In some cases, the method further comprising: obtaining, by theprocessing circuitry, values of one or more situational parametersindicative of a state of the sensor during acquisition of the readings;and wherein the virtual entity visualization is determined also based onthe values of the situational parameters.

In some cases, the situational parameters include one or more of: atemperature of the sensor; a vibration frequency of the sensor; or atype of platform the sensor is connected to.

In accordance with a sixth aspect of the presently disclosed subjectmatter, there is provided a non-transitory computer readable storagemedium having computer readable program code embodied therewith, thecomputer readable program code, executable by at least one processor ofa computer to perform a method comprising: obtaining, by a processingcircuitry, information of one or more virtual entities located within anarea from which readings are acquired by a sensor configured to acquirereadings of real-world data, and display, on an output device, areal-world visualization of the real-world data based on the readings toa user, wherein the sensor has one or more parameters affecting thereal-world visualization, and wherein the information defining, for eachof the virtual entities, one or more simulated physical properties;determining, by the processing circuitry, for at least one given virtualentity of the virtual entities, a virtual entity visualization of thegiven virtual entity, the virtual entity visualization determined bymanipulating a simulated reading of the simulated physical propertiesbased on the parameters affecting the real-world visualization; anddisplaying, by the processing circuitry, the virtual entityvisualizations in combination with the real-world visualization, therebyenabling a user viewing the output device to view the virtual entityvisualizations and the real-world visualization.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to understand the presently disclosed subject matter and to seehow it may be carried out in practice, the subject matter will now bedescribed, by way of non-limiting examples only, with reference to theaccompanying drawings, in which:

FIG. 1 is a schematic illustration of an example environment of a systemfor a mixed reality acquisition device, in accordance with the presentlydisclosed subject matter;

FIG. 2a is a schematic illustration of an example of an output devicedisplaying real-world visualizations and virtual entity visualizationsbefore manipulation of the virtual entity visualizations, in accordancewith the presently disclosed subject matter;

FIG. 2b is a schematic illustration of an example of an output devicedisplaying real-world visualizations and virtual entity visualizationsafter manipulation of the virtual entity visualizations, in accordancewith the presently disclosed subject matter;

FIG. 3a is a schematic illustration of an example of an output devicedisplaying data acquired by a data acquisition device without effects ofthe virtual entities on the data, in accordance with the presentlydisclosed subject matter;

FIG. 3b is a schematic illustration of an example of effects of virtualentities on data acquired by a data acquisition device, in accordancewith the presently disclosed subject matter;

FIG. 3c is a schematic illustration of an example of an output devicedisplaying data acquired by a data acquisition device with effects ofthe virtual entities on the data and virtual entities, in accordancewith the presently disclosed subject matter;

FIG. 4 is a block diagram schematically illustrating one example of amixed reality system, in accordance with the presently disclosed subjectmatter;

FIG. 5 is a flowchart illustrating one example of a sequence ofoperations carried out for virtual entity visualization management, inaccordance with the presently disclosed subject matter;

FIG. 6 is a flowchart illustrating one example of a sequence ofoperations carried out for virtual entity visualization management withactual sensor movement information, in accordance with the presentlydisclosed subject matter;

FIG. 7 is a flowchart illustrating one example of a sequence ofoperations carried out for virtual entity visualization management withsimulated movement information, in accordance with the presentlydisclosed subject matter;

FIG. 8 is a flowchart illustrating one example of a sequence ofoperations carried out for virtual entity visualization management withenvironmental parameters information, in accordance with the presentlydisclosed subject matter;

FIG. 9 is a flowchart illustrating one example of a sequence ofoperations carried out for simulated effects management, in accordancewith the presently disclosed subject matter;

FIG. 10a is a depiction of virtual entity visualization of part of theEiffel tower without taking into consideration the vibration of thesensor;

FIG. 10b is a depiction of the same virtual entity visualization of partof the Eiffel tower where the vibration of the sensor is also used todetermine the virtual entity visualization;

FIG. 11 depicts an image taken of the Eiffel tower by the sensor; and

FIGS. 12a-12e depict examples of prearranged look-up-tables; used todetermine virtual entity visualizations.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of the presentlydisclosed subject matter. However, it will be understood by thoseskilled in the art that the presently disclosed subject matter may bepracticed without these specific details. In other instances, well-knownmethods, procedures, and components have not been described in detail soas not to obscure the presently disclosed subject matter.

In the drawings and descriptions set forth, identical reference numeralsindicate those components that are common to different embodiments orconfigurations.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as obtaining, “determining”,“displaying”, “communication”, “providing” or the like, include actionand/or processes of a computer that manipulate and/or transform datainto other data, said data represented as physical quantities, e.g. suchas electronic quantities, and/or said data representing the physicalobjects. The terms “computer”, “processor”, “processing resource” and“controller” should be expansively construed to cover any kind ofelectronic device with data processing capabilities, including, by wayof non-limiting example, a personal desktop/laptop computer, a server, acomputing system, a communication device, a smartphone, a tabletcomputer, a smart television, a processor (e.g. digital signal processor(DSP), a microcontroller, a field programmable gate array (FPGA), anapplication specific integrated circuit (ASIC), etc.), a group ofmultiple physical machines sharing performance of various tasks, virtualservers co-residing on a single physical machine, any other electroniccomputing device, and/or any combination thereof.

The operations in accordance with the teachings herein may be performedby a computer specially constructed for the desired purposes or by ageneral-purpose computer specially configured for the desired purpose bya computer program stored in a non-transitory computer readable storagemedium. The term “non-transitory” is used herein to exclude transitory,propagating signals, but to otherwise include any volatile ornon-volatile computer memory technology suitable to the application.

As used herein, the phrase “for example,” “such as”, “for instance” andvariants thereof describe non-limiting embodiments of the presentlydisclosed subject matter. Reference in the specification to “one case”,“some cases”, “other cases” or variants thereof means that a particularfeature, structure or characteristic described in connection with theembodiment(s) is included in at least one embodiment of the presentlydisclosed subject matter. Thus, the appearance of the phrase “one case”,“some cases”, “other cases” or variants thereof does not necessarilyrefer to the same embodiment(s).

It is appreciated that, unless specifically stated otherwise, certainfeatures of the presently disclosed subject matter, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the presently disclosed subject matter, which are, forbrevity, described in the context of a single embodiment, may also beprovided separately or in any suitable sub-combination.

In embodiments of the presently disclosed subject matter, fewer, moreand/or different stages than those shown in FIGS. 5-9 may be executed.In embodiments of the presently disclosed subject matter one or morestages illustrated in FIGS. 5-9 may be executed in a different orderand/or one or more groups of stages may be executed simultaneously.FIGS. 1-4, 10-12 illustrate a general schematic of the systemarchitecture in accordance with an embodiment of the presently disclosedsubject matter. Each module in FIGS. 1-4, 10-12 can be made up of anycombination of software, hardware and/or firmware that performs thefunctions as defined and explained herein. The modules in FIGS. 1-4,10-12 may be centralized in one location or dispersed over more than onelocation. In other embodiments of the presently disclosed subjectmatter, the system may comprise fewer, more, and/or different modulesthan those shown in FIGS. 1-4, 1042.

Any reference in the specification to a method should be applied mutatismutandis to a system capable of executing the method and should beapplied mutatis mutandis to a non-transitory computer readable mediumthat stores instructions that once executed by a computer result in theexecution of the method.

Any reference in the specification to a system should be applied mutatismutandis to a method that may be executed by the system and should beapplied mutatis mutandis to a non-transitory computer readable mediumthat stores instructions that may be executed by the system.

Any reference in the specification to a non-transitory computer readablemedium should be applied mutatis mutandis to a system capable ofexecuting the instructions stored in the non-transitory computerreadable medium and should be applied mutatis mutandis to method thatmay be executed by a computer that reads the instructions stored in thenon-transitory computer readable medium.

Bearing this in mind, attention is drawn to FIG. 1, a schematicillustration of an example environment of a system for a mixed realityacquisition device, in accordance with the presently disclosed subjectmatter.

According to the presently disclosed subject matter, a mixed realityenvironment 100 is shown. The mixed reality environment 100 includes oneor more real-world objects (e.g. real-world object A 120-a, real-worldobject B 120-b, real-world object C 120-c, real-world object D 120-d).Real-world objects can be any object or land cover that is physicallypart of mixed reality environment 100. These can be natural or man-madeobjects, for example: trees, hills, clouds, cars, roads, houses or anyother real-world object, land cover and/or weather element.

Additionally, mixed reality environment 100 can include one or morevirtual entities (e.g. virtual entity A 130-a, virtual entity B 130-b,virtual entity C 130-c). Virtual entities can simulate real-worldobjects, land cover and/or weather element but they are not physicallypresent in mixed reality environment 100. Virtual entities can have oneor more properties. These properties can include the location of thevirtual entity. These properties can additionally include information ofthe physical properties of an object simulated by the virtual entity.For example: Virtual entity A 130-a can be simulating a car. Theproperties can include the dimensions of the simulated car, the make ofthe simulated car, the amount of fuel the car has in its fuel tank, etc.The car can also have a location property (e.g. a coordinate, etc.) thatcan be used to establish its virtual geolocation within mixed realityenvironment 100.

In addition, virtual entities (e.g. virtual entity A 130-a, virtualentity B 130-b, virtual entity C 130-c) can cause virtual effects. Thevirtual effects can be determined based on properties of the virtualentities causing the virtual effects. For example, a, virtual entity maybe a heat source (e.g. a fire, an explosion, etc.) that has an effect onthe real-world objects (e.g. real-world object A 120-a, real-worldobject B 120-h, real-world object C 120-c, real-world object D 120-d).These virtual entities can have one or more parameters that can enabledetermining a heat effecting radius (e.g. temperature, material emittingthe heat, etc.

The information regarding the virtual entities (e.g. virtual entity A130-a, virtual entity B 130-b, virtual entity C 130-c) and theirproperties can be stored by a mixed reality system. The mixed realitysystem can be a centralized system, storing the information in a centraldata repository or a distributed system, storing the information in adistributed data repository.

In some cases, the mixed reality system includes one or more dataacquisition devices 140. The data acquisition devices 140 are physicaldevices that are part of the mixed reality environment 100, and are ableto acquire data from at least part of the mixed reality environment 100.The data acquisition device 140 can be a radio receiver (e.g. a VeryHigh Frequency (VHF) radio receiver, a High Frequency (HF) radioreceiver, an Ultra High Frequency (UHF) radio receiver, etc.), a sensor(e.g. a camera, a radar, Night Vision Goggles (NVG), a proximity sensor,temperature sensor, an infrared sensor, pressure sensor, light sensor,touch sensor, ultrasonic sensor, color sensor, humidity sensor, tiltsensor, accelerometer, an acoustic sensor, etc.) or any other devicethat can acquire data from at least part mixed reality environment 100.Acquired real-world data 110 is the data acquired by data acquisitiondevice 140. It is to be noted that, as data acquisition device 140 canonly acquire data that is physically present in environment 100, it doesnot acquire data related to the virtual entities (e.g. virtual entity A130-a, virtual entity B 130-b, virtual entity C 130-c).

In anon-limiting example, data acquisition device 140 can be an infraredsensor and acquired real-world data 110 includes the infrared radiationemitted from at least part of environment 100, as sampled by dataacquisition device 140. These samples include the infrared radiationemitted from real-world object A 120-a and real-world object B 120-bthat are in the part of the environment that is sampled by dataacquisition device 140. These samples do not include infrared radiationemitted from virtual entity A 130-a, even when the geolocation ofvirtual entity A 130-a is within the area of the acquired real-worlddata 110, as virtual entity A 130-a has no physical presence inenvironment 100 and thus does not emit any infrared radiation in thephysical world.

In some cases, the mixed reality system includes one or more outputdevices 150 (e.g. a screen, a Head Mounted Display, a speaker, acombination thereof, etc.). The output device 150 can be used to displaysounds and/or visualizations of the acquired real-world data 110. Themixed reality system can also use the output device to display acombination of sounds and/or visualizations of the acquired real-worlddata 110 (as acquired by data acquisition device 140) and sounds and/orvisualizations of the virtual entities (e.g. virtual entity A 130-a,virtual entity B 130-b, virtual entity C 130-c) that are either locatedwithin an area from which the acquired real-world data 110 readings areacquired or have a virtual effect on the acquired real-world data 110.

The way sounds and/or visualizations of the virtual entities areoutputted to output device 150 by the mixed reality system is dependenton the information regarding the virtual entities (e.g. virtual entity A130-a, virtual entity B 130-b, virtual entity C 130-c) and theirproperties that is stored by the mixed reality system. Continuing ournon-limiting example, virtual entity A 130-a is a car that has aproperty of engine temperature. The visualization of virtual entity A130-a on output device 150 will be an infrared signature that is inaccordance with the engine temperature.

The sounds and/or visualizations of the acquired real-world data 110 canbe affected by one or more parameters of the data acquisition device140, For example: noise model characterizing readings of the dataacquisition device 140, spectral sensitivity of the data acquisitiondevice 140, spectral response of the data acquisition device 140,saturation of the data acquisition device 140, dynamic range of the dataacquisition device 140, dark noise of the data acquisition device 140,signal-to-noise ratio of the data acquisition device 140, detectionlimit of the data acquisition device 140, photo response non uniformityof the data acquisition device 140, penetration depth of the dataacquisition device 140, lens distortion of the data acquisition device140, optical deformation model of the data acquisition device 140,vignetting model of the data acquisition device 140, response todifferent exposure levels of the data acquisition device 140, resolutionmap of the readings acquired by the data acquisition device 140, or anyother parameter of the data acquisition device 140 that affects thesounds and/or visualizations of the acquired real-world data 110.

In addition, the sounds and/or visualizations of the acquired real-worlddata 110 can be affected by a communication link that is used tocommunicate the acquired real-world data 110 from the data acquisitiondevice 140 to the output device 150. Continuing our non-limitingexample, data acquisition device 140 can be an infrared sensor that iscarried on a drone and the acquired real-world data 110 are communicatedvia a communication link to the output device 150 located in a GroundControl Station (GCS) that is remote from the drone carrying theinfrared sensor. The properties of the communication link, such as:bandwidth, delay times, compression rates and methods, etc., can changethe visualization of the acquired real-world data 110 as it is displayedon output device 150. Another example, can be interferences in thecommunication link may change the visualization of the acquiredreal-world data 110 as it is displayed on output device 150.

In addition, the sounds and/or visualizations of the acquired real-worlddata 110 can be affected by one or more algorithms, employed on thereadings of the real-world data by the mixed reality system. The soundsand/or visualizations of the acquired real-world data 110 can also beaffected by other processing done by mixed reality system on theacquired real-world data 110. The processing of the acquired real-worlddata 110 and/or the employing of one or more algorithms on the acquiredreal-world data 110 can be done as part of the acquiring of data by dataacquisition device 140, as part of the communication link, as part ofoutput device 150, or in another processing unit that can be central ordistributed, or in a combination thereof.

FIG. 2a is a non-limiting example of an output device 150 displayingvisualizations of the acquired real-world data 110 and virtual entity(e.g. virtual entity A 130-a, virtual entity B 130-b, virtual entity C130-c) visualizations, before a manipulation of the virtual entityvisualizations by the mixed reality system. FIG. 2a depictsvisualizations of real-world object A′ 120-a′ and real-world object B′120-b′ which are visualizations of real-world object A 120-a andreal-world object B 120-b (respectively) after being affected by (a) oneor more parameters of the data acquisition device 140 affecting thereadings of real-world object A 120-a and real-world object B 120-b,and/or (b) the communication link that is used to communicate thereadings of real-world object A 120-a and real-world object B 120-b fromthe data acquisition device 140 to the output device 150, and/or (c) bythe one or more algorithms and/or other processing employed on thereadings of real-world object A 120-a and real-world object B 120-b.

FIG. 2a also depicts visualizations of virtual entity A 130-a before amanipulation of the visualizations of virtual entity A 130-a by themixed reality system. Before the manipulation of the visualizations ofvirtual entity A 130-a the effects of (a) the one or more parameters ofthe data acquisition device 140 affecting the acquired real-world data110, and/or (b) the one or more algorithms and/or other processingemployed on the readings of the acquired real-world data 110, and/or (c)the communication link that is used to communicate the acquiredreal-world data 110 from the data acquisition device 140 to the outputdevice 150—are not taken into consideration.

This means that before the manipulation of the visualizations of virtualentities, the virtual entities visualizations are output to the outputdevice 150, in a way that is perceived to be different than thereal-world visualizations due to the fact that the virtual entitiesvisualizations are not affected by the same effects that affect thereal-world visualizations.

Continuing our non-limiting example, data acquisition device 140 is aninfrared sensor with a certain noise model parameter, the noise modelcharacterizing readings of the data acquisition device 140. Real-worldobject A′ 120-a′ and real-world object B′ 120-b′ are visualizations ofreal-world object A 120-a and real-world object B 120-b (respectively)as affected by the noise model. Before a manipulation of thevisualization of virtual entity A 130-a by the mixed reality system, thevisualization of virtual entity A 130-a does not reflect the effects ofthe noise model of the data acquisition device 140. In some cases, thiscan result with a user of the mixed reality system, looking at outputdevice 150 and distinguishing a virtual entity A 130-a from real-worldobject A′ 120-a′ and real-world object B′ 120-b′. This can be aproblematic result for mixed reality, systems that aim to seamlesslyintegrate virtual entities (e.g. virtual entity A 130-a, virtual entityB 130-b, virtual entity C 130-c).

It is to be noted that although FIG. 2a exemplifies the problem in thevisual domain, the same problem exists in the sound domain where themixed reality system is required to deal with both real-world sounds andvirtual sounds, and make sure that any manipulation that occurs on thereal-world sounds will take effect also on the virtual sounds.

FIG. 2b continues the non-limiting example of an output device 150displaying visualizations of the acquired real-world data 110 andvirtual entity (e.g. virtual entity A 130-a, virtual entity B 130-b,virtual entity C 130-c) visualizations, after a manipulation of thevirtual entity visualizations by the mixed reality system. Themanipulation of the visualizations of virtual entities takes intoconsideration, by calculating and/or simulating the way that the effects(of (a) the one or more parameters of the data acquisition device 140,and/or (b) the one or more algorithms and/or other processing employedon the readings of the acquired real-world data 110, and/or (c) thecommunication link that is used to communicate the acquired real-worlddata 110 from the data acquisition device 140 to the output device 150)affect the visualizations of virtual entities to produce manipulatedvisualizations of virtual entities.

Continuing our non-limiting example, visualization of virtual entity A′130-a′ is a manipulation of the visualization of virtual entity A 130-aby the mixed reality system, that reflect the effect of the noise modelof the data acquisition device 140, thus a user of the mixed realitysystem, looking at output device 150 will not be able to easily identifythat virtual entity A′ 130-a′ is a virtual entity and not a real-worldobject like real-world object A′ 1204 and real-world object B′ 120-h′.

It is to be noted that although FIG. 2b exemplifies a solution in thevisual domain, the same solution is applicable also to the sound domainwhere the mixed reality system is required to make sure that anymanipulation that occurs on the real-world sounds will take effect alsoon the virtual sounds.

FIG. 3a depicts a non-limiting example of the output device 150displaying the acquired real-world data 110 acquired by the dataacquisition device 140 without the effects of the virtual entities (e.g.virtual entity A 130-a, virtual entity B 130-b, virtual entity C 130-c)on the data. FIG. 3a illustrates the visualizations of the acquiredreal-world data 110 as outputted on output device 150 withoutmanipulating the visualizations of the acquired real-world data 110 inaccordance with simulated effects of the virtual entities. In ournon-limiting example, data acquisition device 140 can be an infraredsensor. Real-world object A 120-a can be a car emitting a first level ofinfrared radiation. Real-world object B 120-b can be a car emitting asecond level of infrared radiation. The infrared sensor samples theinfrared radiation and a visualization relating to the level of sampledinfrared radiation of real-world object A 120-a and real-world object B120-b are displayed on output device 150.

It is to be noted that although FIG. 3a exemplifies the problem in thevisual domain, the same problem exists also in the sound domain wheresounds are not manipulated in accordance with simulated effects of thevirtual entities.

FIG. 3b depicts a non-limiting example of determining, by the mixedreality system, the simulated effects of virtual entities (e.g. virtualentity A 130-a, virtual entity B 130-b, virtual entity C 130-c) on theacquired real-world data 110 acquired by the data acquisition device140, and specifically on the resulting visualization of the acquiredreal-world object A 120-a and real-world object B 120-b.

The mixed reality system needs to take into consideration simulatedeffects of the virtual entities (e.g. virtual entity A 130-a, virtualentity B 130-b, virtual entity C 130-c) on the data acquisition device140 (e.g. a physical sensor, a radio receiver, etc.), and/or on theacquired real-world data 110 and on the resulting visualization of theacquired real-world data 110 as outputted to the output device 150. Thesimulated effects can be determined by the mixed reality system based onproperties of the virtual entities and/or on the properties of the dataacquisition device 140 and/or on the properties of the resultingvisualization of the acquired real-world data 110 as outputted to theoutput device 150. For example, data acquisition device 140 can be aradar, that is part of the mixed reality system. The radar can havetemporary restricted coverage because of an interference from a virtualentity. In another example, data acquisition device 140 can be a thermalsensor blinded (or otherwise affected) by a virtual heat effect comingfrom virtual entities that are in an area viewed by the sensor or arelocated so that they affect the area viewed by the sensor. In yetanother example, data acquisition device 140 is be a real-world radioreceiver that cannot receive radio signals in a given channel due to avirtual entity that is a radio source transmitting a virtual radiosignal in the given channel.

Furthermore, continuing our non-limiting example, virtual entity A130-a, virtual entity B 130-b and virtual entity C 130-c can be heatsources (e.g. fires, explosions, etc.) with certain parameters. Mixedreality system can determine simulated effects (e.g. simulated effect A310-a, simulated effect B 310-b, simulated effect C 310-c) of virtualentity A 130-a virtual entity B 130-b and virtual entity C 130-c withincertain radiuses distance from each of the virtual entities.Specifically, real-world object A 120-a and real-world object B 120-bare affected by the simulated effects (denotated in FIG. 3b by thethickness of the line of real-world object A 120-a and real-world objectB 120-b—the thicker the line the more that part is affected by thesimulated effect), thus the mixed reality system can determine thesimulated infrared radiation emitted from parts of real-world object A120-a and real-world object B 120-b as a consequence of the simulatedeffects. In our example, due to its location with respect to the virtualentities, parts of real-world object B 120-b are affected by bothsimulated effect A 310-a and simulated effect B 310-b. This part issymbolled by a thicker line of real-world object B 120-b.

It is to be noted that although FIG. 3b exemplifies a solution in thevisual domain, the same solution is applicable also to the sound domainwhere the mixed reality system is required to take into accountsimulated effects of the virtual entities on the data acquisition device140 (e.g. a physical sensor, a radio receiver, etc.), and/or on theacquired real-world data 110 and on the resulting visualization of theacquired real-world data 110 as outputted to the output device 150.

FIG. 3c depicts a non-limiting example of the output provided to a userof the mixed reality system on the output device 150, displaying amanipulation of the acquired real-world data 110 acquired by the dataacquisition device 140 with the simulate effects of the virtual entities(e.g. virtual entity A 130-a, virtual entity B 130-h, virtual entity C130-c) on the visualization of the acquired real-world data 110(specifically on the visualizations of real-world object A 120-a andreal-world object B 120-b) and visualization of virtual entity A 130-athat is located within the area of the acquired real-world data 110.Continuing our non-limiting example, the manipulation of thevisualization of real-world object A 120-a and real-world object B 120-bis determined by the mixed realty system based on the heat effects ofthe simulated effects (e.g. simulated effect A 310-a, simulated effect B310-b, simulated effect C 310-c) on the amount of infrared radiationemitting from real-world object A 120-a and real-world object B 120-b.

In addition, the manipulation can be determined based on parameters ofthe data acquisition device 140. Continuing out non-limiting example,the infrared sensor can have a flooding point parameter, determining themaximal amount of infrared radiation the sensor can work with. If thesensor reads an amount of infrared radiation that is higher it isblinded. In our example, the simulated effects (e.g. simulated effect A310-a, simulated effect B 310-b, simulated effect C 310-c) can bring theamount of infrared radiation to blind the sensor and thus the mixedreality system will output a blind-out screen on the output device 150.

Turning to FIG. 4, there is shown is a block diagram schematicallyillustrating one example of a mixed reality system, in accordance withthe presently disclosed subject matter.

According to certain examples of the presently disclosed subject matter,mixed reality system 400 can comprise, or be otherwise associated with,a data repository 430 (e.g. a database, a storage system, a memoryincluding Read Only Memory ROM, Random Access Memory RAM, or any othertype of memory, etc.) configured to store data, including, inter alia;information of one or more virtual entities (e.g. virtual entity A130-a, virtual entity B 130-b, virtual entity C 130-c) including theirproperties, information regarding parameters of the data acquisitiondevice 140, information of one or more algorithms, information ofcommunication links, etc. Data repository 430 can be further configuredto enable retrieval and/or update and/or deletion of the stored data. Itis to be noted that in some cases, data repository 430 can bedistributed, while the mixed reality system 400 has access to theinformation stored thereon, e.g. via a wired or wireless network towhich mixed reality system 400 is able to connect to.

Mixed reality system 400 may further comprise a network interface 420(e.g. a network card, a WiFi client, a LiFi client, 3G/4G client, or anyother network connection enabling component), enabling mixed realitysystem 400 to communicate over a wired or wireless network with one ormore data acquisition devices 140 and/or one or more output devices 150,In some cases, at least one of the connections are over the Internet.

Mixed reality system 400 further comprises a processing circuitry 410.Processing circuitry 410 can be one or more processing units (e.g.central processing units), microprocessors, microcontrollers (e.g.microcontroller units (MCUs)) or any other computing devices or modules,including multiple and/or parallel and/or distributed processing units,which are adapted to independently or cooperatively process data forcontrolling relevant mixed reality system 400 resources and for enablingoperations related to mixed reality system 400 resources.

The processing circuitry 410 can comprise a virtual entity visualizationmanagement module 440 and a simulated effects management module 450.

Virtual entity visualization management module 440 can be configured toperform a virtual entity visualization process, as further detailedherein, inter alia with respect to FIGS. 5-8. Simulated effectsmanagement module 450 can be further configured to perform a simulatedeffects process, as further detailed herein, inter alia with respect toFIG. 9.

Attention is drawn to FIG. 5, a flowchart illustrating one example of asequence of operations carried out for virtual entity visualizationmanagement, in accordance with the presently disclosed subject matter.

According to certain examples of the presently disclosed subject matter,mixed reality system 400 can be configured to perform a virtual entityvisualization process 500, e.g. utilizing the virtual entityvisualization management module 440.

As detailed above, mixed reality system 400 can determine a manipulationon a visualization of virtual entities (e.g. virtual entity A 130-a,virtual entity B 130-b, virtual entity C 130-c) based on parameters of adata acquisition device 140 (e.g. noise model characterizing readings ofthe sensor, spectral sensitivity of the sensor, signal-to-noise ratio ofthe sensor, etc.). These parameters effect the visualization of readingsof acquired real-world data 110, acquired by the data acquisition device140. A non-limiting example, can be that data acquisition device 140 isan infrared sensor and one of its parameters in the noise model of theinfrared sensor.

For this purpose, mixed reality system 400 can be configured to obtaininformation of one or more virtual entities (e.g. virtual entity A130-a, virtual entity B 130-b, virtual entity C 130-c) located within anarea from which the readings are acquired by data acquisition device140, i.e. acquired real-world data 110 (block 510). These reading cancontain readings of real-world objects (e.g. real-world object A 120-areal-world object B 120-b, real-world object C 120-c, real-world objectD 120-d) located within the area from which the readings are acquired.The obtained information defines for each of the virtual entities, oneor more simulated physical properties of the virtual entities. In anon-limiting example, virtual entity A 130-a can be a virtual carlocated within the area from which acquired real-world data 110 isacquired, and a simulated physical property can be the enginetemperature of the car. Mixed reality system 400 can use the informationof the parameters of the data acquisition device 140 and the informationabout the virtual entities and their properties to determine virtualentity visualizations, as further detailed below.

Based on the obtained information, mixed reality system 400 can beconfigured to determine, for at least one given virtual entity of thevirtual entities (e.g. virtual entity A 130-a, virtual entity B 130-b,virtual entity C 130-c), a virtual entity visualization. The virtualentity visualization is determined, by mixed reality system 400, bymanipulating a simulated reading of the simulated physical properties ofthe virtual entity (e.g. one of: virtual entity A 130-a, virtual entityB 130-b, virtual entity C 130-c). The manipulation is determined basedon parameters of data acquisition device 140, affecting the real-worldvisualization (block 520). Mixed reality system 400 is manipulating thevirtual entity visualization so that it will go through a processsimilar to that of the real-world objects (e.g. real-world object A120-a, real-world object B 120-b, real-world object C 120-c, real-worldobject D 120-d) making the virtual entity visualization more realisticto a user of the mixed reality system 400. Continuing our non-limitingexample, the visualization of virtual entity A 120-a, which is a car,will be manipulated to take into consideration the effect of the noisemodel of the data acquisition device 140, which is an infrared sensor.This will make the visualization of the virtual car more realistic whenpresented to the user of mixed reality system 400.

In addition, mixed reality system 400 can manipulate the visualizationof the virtual entities (e.g. virtual entity A 130-a, virtual entity B130-b, virtual entity C 130-c) to take into consideration also one ormore algorithms, employed on the readings of the real-world data. Anon-limiting example of an algorithm can be an image sharpeningalgorithm. In order for the visualization of the virtual entity to lookas closest to a real-world object (e.g. real-world object A 120-a,real-world object B 120-b, real-world object C 120-c, real-world objectD 120-d), the same or similar algorithm employed on real-world objectsis employed on the virtual entities.

In some cases, the parameters of data acquisition device 140 affect thereal-world visualization by affecting the readings of the acquiredreal-world data 110. In these cases, the parameters can include one ormore of: a noise model characterizing readings of the sensor, a spectralsensitivity of the sensor, a spectral response of the sensor, asaturation of the sensor, a dynamic range of the sensor, a dark noise ofthe sensor, a signal-to-noise ratio of the sensor, a detection limit ofthe sensor, a photo response non uniformity of the sensor, a penetrationdepth of the sensor, a lens distortion of the sensor, an opticaldeformation model of the sensor, a vignetting model of the sensor,response to different exposure levels of the sensor, a resolution map ofthe readings acquired by the sensor, etc.

In some cases, mixed reality system 400 can manipulate the visualizationof the virtual entities (e.g. virtual entity A 130-a, virtual entity B130-b, virtual entity C 130-c) by partially employing the processes, thealgorithms and/or the parameters of the data acquisition device 140 thataffect the real-world visualization.

In some cases, mixed reality system 400 can manipulate the visualizationof the virtual entities (e.g. virtual entity A 130-a, virtual entity B130-b, virtual entity C 130-c) to imitate visualization of readings ofthe acquired real-world data 110. This can be done not by putting thevisualization of the virtual entities through a process similar to thatof the real-world objects (e.g. real-world object A 120-a, real-worldobject B 120-b, real-world object C 120-c, real-world object D 120-d),but by analyzing the visualization of readings of the acquiredreal-world data 110 and manipulate the visualization of the virtualentities to imitate them. For example: if visualization of readings ofthe acquired real-world data 110 are darker at night, mixed realitysystem 400 can manipulate the visualization of the virtual entities toseem darker at night time as well.

Mixed reality system 400 can be configured to display the virtual entityvisualizations in combination with the real-world visualization. Thisallows the user of mixed reality system 400 viewing the output device150 to see the manipulated virtual entity visualizations and thereal-world visualization together (block 530). Continuing ournon-limiting example, output device 150 can display a visualization of areal-world car (as actually acquired by the sensor) and a visualizationof the virtual car, manipulated to take into consideration the effectsof the parameters of the sensor. This will create a visualization of thevirtual entity (e.g. one of: virtual entity A 130-a, virtual entity B130-b, virtual entity C 130-c) as close as possible to visualizations ofthe real-world objects (e.g. real-world object A 120-a, real-worldobject B 120-b, real-world object C 120-c, real-world object D 120-d).In some cases, the user of mixed reality system 400 will not be able toeasily identify that virtual car is a virtual entity and not areal-world car.

In some cases, the virtual entity visualizations are displayed by (a)injecting the virtual entity visualizations into the output device 150,or (b) by displaying the virtual entity visualizations on a see-throughoutput device overlaying the output device 150, or (c) by projecting thevirtual entity visualizations on the real-world visualization. In thesecases, the virtual entity visualizations are provided to the outputdevice 150 through a communication channel that is different from thecommunication channel through which the real-world visualizations areprovided to the output device 150. Thus, the virtual entityvisualizations are not affected as the real-world visualizations areaffected. It is to be noted that in some cases the acquired real-worlddata 110 undergoes processing during its communication from the dataacquisition device 140 to the output device 150 through communicationchannel.

The resulting virtual entity visualization will be perceived closer to avisualization of a given virtual entity had it been a real-world entityhaving physical properties similar to the simulated physical propertiesthan an alternative visualization of the same given virtual entitygenerated without manipulating the simulated reading of the simulatedphysical properties based on the parameters affecting, the readings ofthe data acquisition device 140.

It is to be noted that, with reference to FIG. 5, some of the blocks canbe integrated into a consolidated block or can be broken down to a fewblocks and/or other blocks may be added. Furthermore, in some cases, theblocks can be performed in a different order than described herein. Itis to be further noted that some of the blocks are optional. It shouldbe also noted that whilst the flow diagram is described also withreference to the system elements that realizes them, this is by no meansbinding, and the blocks can be performed by elements other than thosedescribed herein.

Attention is drawn to FIG. 6, a flowchart illustrating one example of asequence of operations carried out for virtual entity visualizationmanagement with actual sensor movement information, in accordance withthe presently disclosed subject matter.

According, to certain examples of the presently disclosed subjectmatter, mixed reality system 400 can be configured to perform a virtualentity visualization process with actual sensor movement information600, e.g. utilizing the virtual entity visualization management module440.

As detailed above, mixed reality system 400 can determine a manipulationon a visualization of virtual entities (e.g. virtual entity A 130-a,virtual entity B 130-b, virtual entity C 130-c) based on parameters of adata acquisition device 140. Mixed reality system 400 can furtherdetermine the manipulation also based on information of movement of thedata acquisition device 140.

For this purpose, mixed reality system 400 can be configured, beforeperformance of block 530 of virtual entity visualization process 500, tofurther obtain information of movement of the data acquisition device140 during acquisition of the readings (block 610). It is to be notedthat the movement of the data acquisition device 140 can be caused insome cases by movement of the data acquisition device 140 itself and/orby a movement of a platform to which data acquisition device 140 isconnected thereto. In a non-limiting example, data acquisition device140 can be a camera. In case the camera moves during data acquisition,the information about the movement is obtained by mixed reality system400, The movement can be caused by an operator moving the camera itselfor rotating the camera, thus changing the camera's viewing angel, or inother cases, by an operator moving a platform to which the camera isconnected. In some cases, the camera may be connected to a movingplatform and the camera itself can move while the platform is alsomoving.

After obtaining the information of movement of the data acquisitiondevice 140, if any, mixed reality system 400 can be configured todetermine if the movement of the data acquisition device 140 generates amovement effect on the real-world visualization, utilizing theinformation and the parameters.

The movement effect can include effects on an image the sensor produces.This image effect happens when the rate of movement of the dataacquisition device 140 is higher than the image actuation rate, thus theobjects in the image be blurred or if a video is created the objects canseem to “jump” between frames, as the rate of change of the line ofsight of the data acquisition device 140 is higher than the renderingrate of the video. In addition, movement effects can include in additioneffects issuing from vibrations of the data acquisition device 140 inone or more axis.

In some cases, the movement effect can be determined by utilizinglook-up-tables that determine a transformation function to be used todetermine the movement effect for each value range of movement of thesensor.

If so determined, mixed reality system 400, can further manipulate thesimulated reading to further include simulating the movement effect onthe simulated reading (block 620). This determination can be achieved byutilizing a simulated model of the data acquisition device 140,simulating how data acquisition device 140 is affected by its ownmovement.

Continuing our non-limiting example, the movement of the camera duringdata acquisition creates an effect. Mixed reality system 400 willmanipulate visualizations of virtual entities (e.g. virtual entity A130-a, virtual entity B 130-b, virtual entity C 130-c) in the area ofthe acquired real-world data 110 to simulate the movement effect. Insome cases, mixed reality system 400 calculates the movement effect forthe visualizations of all virtual entities (e.g. virtual entity A 130-a,virtual entity B 130-b, virtual entity C 130-c) and can employ theeffect on the visualizations of virtual entities that are in the area ofthe acquired real-world data 110 to simulate the movement effect. Thus,a user of mixed reality system 400 will perceive the virtualvisualization as he would perceive a similar real-world object (e.g.real-world object A 120-a, real-world object B 120-b, real-world objectC 120-c, real-world object D 120-d).

It is to be noted that, with reference to FIG. 6, some of the blocks canbe integrated into a consolidated block or can be broken down to a fewblocks and/or other blocks may be added. Furthermore, in some cases, theblocks can be performed in a different order than described herein. Itis to be further noted that some of the blocks are optional. It shouldbe also noted that whilst the flow diagram is described also withreference to the system elements that realizes them, this is by no meansbinding, and the blocks can be performed by elements other than thosedescribed herein.

FIG. 7 is a flowchart illustrating one example of a sequence ofoperations carried out for virtual entity visualization management withsimulated movement information, in accordance with the presentlydisclosed subject matter.

According to certain examples of the presently disclosed subject matter,mixed reality system 400 can be configured to perform a virtual entityvisualization process with simulated movement information 700, e.g.utilizing the virtual entity visualization management module 440.

As detailed above, mixed reality system 400 can determine a manipulationon a visualization of virtual entities (e.g. virtual entity A 130-a,virtual entity B 130-h, virtual entity C 130-c) based on parameters of adata acquisition device 140. Mixed reality system 400 can furtherdetermine the manipulation also based on information of simulatedmovement of the given virtual entity (e.g. one of: virtual entity A130-a, virtual entity B 130-b, virtual entity C 130-c) duringacquisition of the readings by data acquisition device 140.

For this purpose, mixed reality system 400 can be configured, beforeperformance of block 530 of virtual entity visualization process 500, tofurther obtain information of simulated movement of the given virtualentity (e.g. one of: virtual entity A 130-a, virtual entity B 130-b,virtual entity C 130-c) during acquisition of the readings (block 710).In a non-limiting example, data acquisition device 140 can be a cameraand virtual entity A 130-a is a car that can be determined, according tothe parameters of the camera and/or according to the parameters of thevirtual car, to be moving during the acquisition of the readings by thedata acquisition device 140. The information about the simulatedmovement of virtual entity A 130-a is obtained by mixed reality system400.

After obtaining the information of the simulated movement of the givenvirtual entity (e.g. one of: virtual entity A 130-a, virtual entity B130-b, virtual entity C 130-c), if any, mixed reality system 400 can beconfigured to determine if the simulated movement of the given virtualentity is required to generate an effect on the simulated reading,utilizing the information and the parameters. If so determined, mixedreality system 400, can further manipulate the simulated reading tofurther include the simulated effect (block 720).

In some cases, the determination if the simulated movement of the givenvirtual entity is required to generate an effect on the simulatedreading can be done by utilizing look-up-tables that determine atransformation function to be used to determine the effect for eachvalue range of movement of the given virtual entity.

The determination if the simulated movement of the given virtual entityis required to generate an effect on the simulated reading can be doneby utilizing a simulation model. The simulation model simulates how dataacquisition device 140 behaves when acquired real-world data 110 ismoving while acquired by data acquisition device 140. The simulatedmovement of the given virtual entity is used together with thesimulation model to determine the effect on the simulated reading. Forexample: data acquisition device 140 can be a camera. When this cameraacquires real-world data 110 that is moving, the movement produces asmear effect on the visualizations of the real-world objects (e.g.real-world object A 120-a, real-world object B 120-b, real-world objectC 120-c, real-world object D 120-d) by the camera. Mixed reality system400 can use a simulation model of the camera to create the same smeareffect on the simulated readings. In another non-limiting example, dataacquisition device 140 can be a radar and the radar lock can be lostwhen the tracked object is in movement. Another non-limiting exampleacquisition device 140 can be a datalink, and the communication devicescan suffer from limited communication or cutoff from the datalink due totheir movement.

Continuing our non-limiting example, the movement of the virtual carduring data acquisition creates a simulated effect, Mixed reality system400 can manipulate visualizations of virtual entities (e.g. virtualentity A 130-a, virtual entity B 130-b, virtual entity C 130-c) in thearea of the acquired real-world data 110 to include the simulatedeffect. Thus, a user of mixed reality system 400 will perceive thevirtual visualization as he would perceive a similar real-world object(e.g. real-world object A 120-a, real-world object B 120-b, real-worldobject C 120-c, real-world object D 120-d). It is to be noted that, withreference to FIG. 7, some of the blocks can be integrated into aconsolidated block or can be broken down to a few blocks and/or otherblocks may be added. Furthermore, in some cases, the blocks can beperformed in a different order than described herein. It is to befurther noted that some of the blocks are optional. It should be alsonoted that whilst the flow diagram is described also with reference tothe system elements that realizes them, this is by no means binding, andthe blocks can be performed by elements other than those describedherein.

Attention is drawn to FIG. 8, a flowchart illustrating one example of asequence of operations carried out for virtual entity visualizationmanagement with environmental parameters information, in accordance withthe presently disclosed subject matter.

According to certain examples of the presently disclosed subject matter,mixed reality system 400 can be configured to perform a virtual entityvisualization process with environmental parameters information 800,e.g. utilizing the virtual entity visualization management module 440.

As detailed above, mixed reality system 400 can determine a manipulationon a visualization of virtual entities (e.g. virtual entity A 130-a,virtual entity B 130-b, virtual entity C 130-c) based on parameters of adata acquisition device 140. Mixed reality system 400 can furtherdetermine the manipulation also based on information of environmentalparameters of at least part of a region from which acquired real-worlddata 110 is acquired, during acquisition of the readings by dataacquisition device 140.

For this purpose, mixed reality system 400 can be configured, beforeperformance of block 530 of virtual entity visualization process 500, tofurther obtain information of environmental parameters of at least partof a region from which the readings are acquired during acquisition ofthe readings (block 810). In a non-limiting example, data acquisitiondevice 140 can be a camera and the environmental parameters can be thetemperature in the region from which the readings are acquired at thetime of acquisition of the readings.

After obtaining the information of the environmental parameters, mixedreality system 400 can be configured to determine if the environmentalparameters generate an environmental effect on the real-worldvisualization, utilizing the information and the parameters. If sodetermined, mixed reality system 400, can further manipulate thesimulated reading to further include simulating the environmental effect(block 820). Continuing our non-limiting example, mixed reality system400 will manipulate visualizations of virtual entities (e.g. virtualentity A 130-a, virtual entity B virtual entity C 130-c) in the area ofthe acquired real-world data 110 to include an environmental effect ofthe temperature on the way he camera works. Thus, a user of mixedreality system 400 will perceive the virtual visualization as he wouldperceive a similar real-world object (e.g. real-world object A 120-a,real-world object B 120-b, real-world object C 120-c, real-world objectD 120-d).

According to certain examples of the presently disclosed subject matter,mixed reality system 400 can be further configured to obtain values ofone or more situational parameters indicative of a state of the sensorduring acquisition of the readings. The virtual entity visualization canbe determined also based on the values of the situational parameters.

In these cases, data acquisition device 140 can be a sensor and thesituational parameters can include a temperature of the sensor and/orits surroundings, a vibration frequency of the sensor, a type ofplatform the sensor is connected to, a velocity of the platform thesensor is connected to, a topography the platform the sensor isconnected to is in; or any other parameter indicative of the state ofthe sensor during acquisition of the readings.

The situational parameters can be readings of one or more situationalsensors, sensing the state of the sensor and/or its surroundings. Forexample: a temperature situational sensor can sense the temperature ofthe sensor and/or its surroundings.

Another example can be an Inertial Measurement Unit (MU) situationalsensor that can sense the vibration frequency of the sensor and/or thevibration of the sensor within a coordination system. In this case, thevirtual entity visualization can be determined according to vibrationfrequency and/or vibration of the sensor within a coordination systeminformation in order to imitate real-world objects visualizations. In anon-limiting example, wherein the sensor is a camera, FIG. 10a is adepiction of virtual entity visualization of part of the Eiffel towerwithout taking into consideration the vibration of the sensor. FIG. 10bis a depiction of the same virtual entity visualization of part of theEiffel tower where the vibration of the sensor is also used to determinethe virtual entity visualization.

In some cases, at least part of the situational parameters can bededucted from analysis of readings of the sensor. In a non-limitingexample, wherein the sensor is a camera, FIG. 11 depicts an image takenof the Eiffel tower by the sensor. The mixed reality system 400 cancalculate distortions based on image analysis and deduce at least someof the situational parameters. The virtual entity visualizations canthen be determined also based on the deduced situational parameters.

A non-limiting example of the way the situational parameters are alsoused to determine the virtual entity visualizations can be by usingprearranged look-up-tables. These look-up-tables can be determinedduring calibration process of the sensor or based on the sensor'smanufacturer data. FIG. 12a depicts a non-limiting example of alook-up-table for temperature distortions of the sensor. For eachmeasured temperature of the sensor and/or its surroundings, thelook-up-table entry is associated with a transformation function(transformation functions A, B or C) used to determine the entityvisualization. FIG. 12h depicts a non-limiting example of alook-up-table for types of topographies the platform the sensor isconnected to is in and the transformation functions associated with eachtype of topography. FIG. 12c depicts a non-limiting example of alook-up-table for values of measured velocity of the sensor (or of theplatform the sensor is connected to) and the transformation functionsassociated with the velocity readings. FIG. 12d depicts a non-limitingexample of a look-up-table for types and sub-types of platforms thesensor is connected to and the transformation functions associated witheach type and sub-type of platform.

A non-limiting example of using multiple look-up-tables to determine thevirtual entity visualizations is depicted in FIG. 12e . In the exampleof FIG. 12c the sensor is connected to a platform of type “APC” and thesensor type is “CMOS type” and the velocity of the sensor is “50” andthe topography where the platform the sensor is connected to is a “Road”and the temperature of the sensor is “10” thus multiple distortionfunction (in this example: distortion function LL, L, M, AB and B) areused to determine the virtual entity visualizations.

It is to be noted that, with reference to FIG. 8, some of the blocks canbe integrated into a consolidated block or can be broken down to a fewblocks and/or other blocks may be added. Furthermore, in some cases, theblocks can be performed in a different order than described herein. Itis to be further noted that some of the blocks are optional. It shouldbe also noted that whilst the flow diagram is described also withreference to the system elements that realizes them, this is by no meansbinding, and the blocks can be performed by elements other than thosedescribed herein.

It is to be noted that in some cases, mixed reality system 400 can befurther configured to obtain values of one or more situationalparameters indicative of a state of the data acquisition device 140during acquisition of the readings. The virtual entity visualization canbe determined also based on the values of the situational parameters.This can be done by determining situational effects on the readings ofthe data acquisition device 140 based on the situational parameters. Forexample: when the data acquisition device 140 is vibrating, the readingswill be affected accordingly. The determination of the virtual entityvisualization takes the situational effects on the data acquisitiondevice 140 into account.

In some cases, data acquisition device 140 can be a sensor and thesituational parameters can include a temperature of the sensor and/orits surroundings, a vibration frequency of the sensor, a type ofplatform the sensor is connected to, a velocity of the platform thesensor is connected to, a topography the platform the sensor isconnected to is in; or any other parameter indicative of the state ofthe sensor during acquisition of the readings.

The situational parameters can be readings of one or more situationalsensors, sensing the state of the sensor and/or its surroundings. Forexample: a temperature situational sensor can sense the temperature ofthe sensor and/or its surroundings.

An example can be an Inertial Measurement Unit (IMU) situational sensorthat can sense the vibration frequency of the sensor and/or thevibration of the sensor within a coordination system (e.g. the vibrationof the sensor in length (X plane), height (Y plane) and depth (7 plane)planes).

In some cases, the vibrations of the sensor are caused by the movementof the platform the sensor is installed on. For example: a camerainstalled on a moving vehicle can vibrate because of the movement of thevehicle.

The IMU can be part of the sensor itself and/or installed on the sensor.The IMU can alternatively be installed on the platform that the sensoris connected to, for example: the sensor is installed on a vehicle andthe IMU is installed on the same vehicle in a location allowing the IMUto sense the vibration of the sensor. In other cases, the IMU can beinstalled in a location that allows it to sense the vibration of thesensor, for example: installed on a second platform, different from theplatform that the sensor is installed on, that is in a proximity of thesensor, allowing for the measurement of the sensor's vibrations by theIMU. In these cases, the virtual entity visualization can be determinedaccording to vibration frequency and/or vibration measurement of thesensor within a coordination system in order to imitate real-worldobjects visualizations. When the IMU senses the sensor's field-of-viewvibrating in a certain X-Y-Z measurement, this certain vibration will beprojected on the virtual entity visualization by determining it to bevibrating in the same X-Y-Z measurement.

In a non-limiting example, wherein the sensor is a camera, FIG. 10a is adepiction of a scene that is in the field-of-view acquired by thecamera. In our example, a scene with the Eiffel tower in the background.The sensor is vibrating and an IMU is measuring the vibration. FIG. 10bincludes a depiction of the same scene depicted in FIG. 10a , but theimage is distorted due to the vibrations of the camera. FIG. 10b alsoincludes a virtual entity visualization 1010, which in our example is avirtual cross, that is overlaid on the scene. Virtual entityvisualization 1010 is overlaid on the Eiffel tower scene without takinginto consideration the vibration of the sensor. FIG. 10c is a depictionof the same scene, wherein virtual entity visualization 1020 is thevisualization of virtual entity visualization 1010 determined by takinginto consideration the vibration of the sensor. For example, when thecamera is vibrating in the X and Y planes, the properties of thevibrations are sensed by the IMU and applied to the visualization ofvirtual entity visualization 1020 in order to imitate real lifeconditions, thus virtual entity visualization 1020 is a manipulation onvirtual entity visualization 1010 to include the effect of the vibrationof the camera in the X and Y planes.

In some cases, at least part of the situational parameters can bededucted from analysis of readings of the sensor. In a non-limitingexample, wherein the sensor is a camera, FIG. 11 depicts an image takenof the Eiffel tower by a vibrating sensor. The mixed reality system 400can calculate distortions based on image analysis and deduce at leastsome of the situational parameters. In our example, a smallregion-of-interest 1110 out of the entire scene is analyses to indicatethe level of vibrations by calculating gradient properties that indicatethe sharpness/blurry level of the image within region-of-interest 1110.The virtual entity visualizations can then be determined also based onthe deduced situational parameters and generated accordingly. In ourexample, the virtual entity visualizations are generated using the samegradient parameters deducted from the analysis of region-of-interest1110.

It is to be noted that other image analysis methods are also available.A non-limiting example of the way the situational parameters are alsoused to determine the virtual entity visualizations can be by usingprearranged look-up-tables. These look-up-tables can be determinedduring calibration process of the sensor or based on the sensor'smanufacturer data. FIG. 12a depicts a non-limiting example of alook-up-table for temperature distortions of the sensor. For eachmeasured temperature of the sensor and/or its surroundings, thelook-up-table entry is associated with a transformation function(transformation functions A, B, C or D) used to determine the wrappingsand/or distortions of the virtual entity visualization. A non-limitingexample can be of a measured temperature of the sensor of 20 degreesCelsius, thus the virtual entity visualizations will be generated usingtransformation function D. FIG. 12b depicts a non-limiting example of alook-up-table for types of topographies the platform the sensor isconnected to is in and the transformation functions associated with eachtype of topography. FIG. 12c depicts a non-limiting example of alook-up-table for values of measured velocity of the sensor (or of theplatform the sensor is connected to) and the transformation functionsassociated with the velocity readings. FIG. 12d depicts a non-limitingexample of a look-up-table for types and sub-types of platforms thesensor is connected to and the transformation functions associated witheach type and sub-type of platform.

A non-limiting example of using multiple look-up-tables to determine thevirtual entity visualizations is depicted in FIG. 12e , In the exampleof FIG. 12e the sensor is connected to a platform of type “APC” and thesensor type is “CMOS type” and the velocity of the sensor is “50” andthe topography where the platform the sensor is connected to is a “Road”and the temperature of the sensor is “10” thus multiple distortionfunction (in this example: distortion functions L, M, AB and B) are usedtogether to determine the virtual entity visualizations.

Attention is drawn to FIG. 9, a flowchart illustrating one example of asequence of operations carried out for simulated effects management, inaccordance with the presently disclosed subject matter.

According to certain examples of the presently disclosed subject matter,mixed reality system 400 can be configured to perform a simulatedeffects process 900, e.g. utilizing the simulated effects managementmodule 450.

As detailed above, mixed reality system 400 can take into considerationsimulated effects of the virtual entities (e.g. virtual entity A 130-a,virtual entity B 130-b, virtual entity C 130-c) on the data acquisitiondevice 140 (e.g. a physical sensor, a radio receiver, etc.), on theacquired real-world data 110 and/or on the resulting sounds and/orvisualization of the acquired real-world data 110 as outputted to theoutput device 150. The simulated effects can be determined by the mixedreality system based on properties of the virtual entities. In anon-limiting example, data acquisition device 140 can be a thermalsensor blinded (or otherwise affected) by a virtual heat effect comingfrom virtual entities that are in an area viewed by the sensor or arelocated so that they affect the area viewed by the sensor.

For this purpose, mixed reality system 400 can be configured to obtaindata acquired by the data acquisition device 140, and information of oneor more virtual entities (e.g. virtual entity A 130-a, virtual entity B130-b, virtual entity C 130-c). These virtual entities have propertiesenabling mixed reality system 400 to determine the simulated effects ofthe virtual entities on the data (block 910). Continuing ournon-limiting example, the obtained data is the acquired real-world data110 acquired by the infrared sensor. The information is of virtual firesand their properties. These virtual fires are affecting the acquiredreal-world data 110.

It is to be noted that data acquisition device 140 can be a sensor or aradio receiver. In case data acquisition device 140 is a sensor, it canbe one of the following a camera, a radar, Night Vision Goggles (NVG), aproximity sensor, temperature sensor, an infrared sensor, pressuresensor, light sensor, touch sensor, ultrasonic sensor, color sensor,humidity sensor, tilt sensor, accelerometer, an acoustic sensor, or anyother device with sensing capabilities.

It is to be noted that the simulated effects can be one or more of:virtual heat, virtual light, virtual touch, virtual shade, virtualsound, virtual topography, virtual smoke, virtual hit, virtual ice orany other effect.

In some cases, the data is acquired from a training environment, and thevirtual entities (e.g. virtual entity A 130-a, virtual entity B 30-b,virtual entity C 130-c) are designed to simulate training scenarios.

After obtaining the data and the information, mixed reality system 400can be further configured to determine the simulated effects of thevirtual entities (e.g. virtual entity A 130-a, virtual entity B 130-b,virtual entity C 130-c) on the data, utilizing the properties of thevirtual entities (block 920). The determination of the simulated effectsof the virtual entities (e.g. virtual entity A 130-a, virtual entity B130-b, virtual entity C 130-c) on the data can be done by utilizing asimulated model, simulating the way data acquisition device 140 isaffected by the simulated effects, for example: data acquisition device140 can be a thermal sensor that is “blinded” by a near-by virtualexplosion, thus the output device 150 will be blank and display no data.Another example can be of a radio device that has less data capacity forreceiving and transmitting data due to simulated effects of virtualentities and the data they create. In this example the radio device datarate will go down due to the load on the network created by virtualentities. In addition, the determination can be done by image analysisof an image displayed on output device 150. The analysis determines thereal-world objects displayed and using the simulated model to determinethe simulated effect on the data.

Continuing our non-limiting example, mixed reality system 400 determinesthe simulated effects of the virtual fire has on the acquired real-worlddata 110. In our example, the virtual fires simulate effect is toincrease the amount of infrared radiation sampled from the acquiredreal-world data 110.

In addition, mixed reality system 400 can determine the simulatedeffects also utilizing one or more parameters of the sensor or the radioreceiver. In case, the data acquisition device 140 is a radio receiver,the parameter can also be a load parameter that effects the data rate ofthe radio receiver. For example: a large number of virtual entities(e.g. virtual entity A 130-a, virtual entity B 130-b, virtual entity C130-c) can increase the load and lower the data rate of the radioreceiver.

Mixed reality system 400 can then be further configured to provide theuser with output on the output device being a manipulation of the datareflecting the simulated effects (block 930). Continuing ournon-limiting example, mixed reality system 400 can determine thesimulated effects of the virtual fires have blinded the data acquisitiondevice 140, and only a blind-out screen will be presented to the user ofmixed reality system 400.

It is to be noted that, with reference to FIG. 9, some of the blocks canbe integrated into a consolidated block or can be broken down to a fewblocks and/or other blocks may be added. Furthermore, in some cases, theblocks can be performed in a different order than described herein. Itis to be further noted that some of the blocks are optional. It shouldbe also noted that whilst the flow diagram is described also withreference to the system elements that realizes them, this is by no meansbinding, and the blocks can be performed by elements other than thosedescribed herein.

It is to be understood that the presently disclosed subject matter isnot limited in its application to the details set forth in thedescription contained herein or illustrated in the drawings. Thepresently disclosed subject matter is capable of other embodiments andof being practiced and carried out in various ways. Hence, it is to beunderstood that the phraseology and terminology employed herein are forthe purpose of description and should not be regarded as limiting. Assuch, those skilled in the art will appreciate that the conception uponwhich this disclosure is based may readily be utilized as a basis fordesigning other structures, methods, and systems for carrying out theseveral purposes of the present presently disclosed subject matter.

It will also be understood that the system according to the presentlydisclosed subject matter can be implemented, at least partly, as asuitably programmed computer. Likewise, the presently disclosed subjectmatter contemplates a computer program being readable by a computer forexecuting the disclosed method. The presently disclosed subject matterfurther contemplates a machine-readable memory tangibly embodying aprogram of instructions executable by the machine for executing thedisclosed method.

1. A mixed reality system, comprising: a data acquisition deviceconfigured to acquire real-world data; an output device for providingthe real-world data to a user; and a processing circuitry configured to:obtain (a) the real-world data acquired by the data acquisition device,and (b) information of one or more virtual entities having propertiesenabling determination of simulated effects of the virtual entities onthe real-world data; determine the simulated effects of the virtualentities on the data utilizing the properties and one or more parametersof the data acquisition device, wherein the one or more parametersdefining effects of the virtual entities on a behavior of the dataacquisition device; and provide the user with output on the outputdevice being a manipulation of the real-world data reflecting thesimulated effects.
 2. The mixed reality system of claim 1, wherein thedata acquisition device is a sensor or a radio receiver.
 3. The mixedreality system of claim 2, wherein the sensor is one of the following: acamera, a radar, Night Vision Goggles (NVG), a proximity sensor,temperature sensor, an infrared sensor, pressure sensor, light sensor,touch sensor, ultrasonic sensor, color sensor, humidity sensor, tiltsensor, accelerometer, or an acoustic sensor.
 4. (canceled)
 5. The mixedreality system of claim 1, wherein the real-world data is acquired froma training environment, and wherein the virtual entities are designed tosimulate training scenarios.
 6. The mixed reality system of claim 1,wherein the simulated effect is one or more of: virtual heat, virtuallight, virtual touch, virtual shade, virtual sound, virtual topography,virtual smoke, virtual hit, or virtual ice.
 7. A method comprising:obtaining, by a processing resource, (a) real-world data acquired by adata acquisition device configured to acquire real-world data, and (b)information of one or more virtual entities having properties enablingdetermination of simulated effects of the virtual entities on thereal-world data; determining, by the processing resource, the simulatedeffects of the virtual entities on the real-world data utilizing theproperties and one or more parameters of the data acquisition device,wherein the one or more parameters defining effects of the virtualentities on a behavior of the data acquisition device; and providing, bythe processing resource, a user of an output device, used for providingthe real-world data to the user, with output on the output device beinga manipulation of the real-world data reflecting the simulated effects.8. The method of claim 7, wherein the data acquisition device is asensor or a radio receiver.
 9. The method of claim 8, wherein the sensoris one of the following: a camera, a radar, Night Vision Goggles (NVG),a proximity sensor, temperature sensor, an infrared sensor, pressuresensor, light sensor, touch sensor, ultrasonic sensor, color sensor,humidity sensor, tilt sensor, accelerometer, or an acoustic sensor. 10.(canceled)
 11. The method of claim 7, wherein the real-world data isacquired from a training environment, and wherein the virtual entitiesare designed to simulate training scenarios.
 12. The method of claim 7,wherein the simulated effect is one or more of: virtual heat, virtuallight, virtual touch, virtual shade, virtual sound, virtual topography,virtual smoke, virtual hit, or virtual ice.
 13. A non-transitorycomputer readable storage medium having computer readable program codeembodied therewith, the computer readable program code, executable by atleast one processor of a computer to perform a method comprising:obtaining, by a processing resource, (a) real-world data acquired by adata acquisition device configured to acquire real-world data, and (b)information of one or more virtual entities having properties enablingdetermination of simulated effects of the virtual entities on thereal-world data; determining, by the processing resource, the simulatedeffects of the virtual entities on the real-world data utilizing theproperties and one or more parameters of the data acquisition device,wherein the one or more parameters defining effects of the virtualentities on a behavior of the data acquisition device; and providing, bythe processing resource, a user of an output device, used for providingthe real-world data to the user, with output on the output device beinga manipulation of the real-world data reflecting the simulated effects.14.-46. (canceled)